Approximate Solution Techniques for Factored First-Order MDPs

Most traditional approaches to probabilistic planning in relationally specified MDPs rely on grounding the problem w.r.t. specific domain instantiations, thereby incurring a combinatorial blowup in the representation. An alternative approach is to lift a relational MDP to a firstorder MDP (FOMDP) specification and develop solution approaches that avoid grounding. Unfortunately, state-of-the-art FOMDPs are inadequate for specifying factored transition models or additive rewards that scale with the domain size—structure that is very natural in probabilistic planning problems. To remedy these deficiencies, we propose an extension of the FOMDP formalism known as a factored FOMDP and present generalizations of symbolic dynamic programming and linear-value approximation solutions to exploit its structure. Along the way, we also make contributions to the field of first-order probabilistic inference (FOPI) by demonstrating novel first-order structures that can be exploited without domain grounding. We present empirical results to demonstrate that we can obtain solutions whose complexity scales polynomially in the logarithm of the domain size—results that are impossible to obtain with any previously proposed solution method. Introduction There has been a great deal of research in recent years aimed at exploiting structure in order to compactly represent and efficiently solve decision-theoretic planning problems in the Markov decision process (MDP) framework (Boutilier, Dean, & Hanks 1999). While traditional approaches to solving MDPs typically used an enumerated state and action model, this approach has proven impractical for large-scale AI planning tasks where the number of distinct states in a model can easily exceed the limits of primary and secondary storage on modern computers. Fortunately, many MDPs can be compactly described in a propositionally factored model that exploits various independences in the reward and transition functions. And not only can this independence be exploited in the problem representation, it can often be exploited in exact and approximate solution methods as well (Hoey et al. 1999; Copyright c © 2007, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. St-Aubin, Hoey, & Boutilier 2000; Guestrin et al. 2002). However, while recent techniques for factored MDPs have proven effective, they cannot generally exploit first-order structure. Yet many realistic planning domains are best represented in first-order terms, exploiting the existence of domain objects, relations over these objects, and the ability to express objectives and action effects using quantification. These deficiencies have motivated the development of the first-order MDP (FOMDP) framework (Boutilier, Reiter, & Price 2001) that directly exploits the first-order representation of MDPs to obtain domain-independent solutions. While FOMDP approaches have demonstrated much promise (Kersting, van Otterlo, & de Raedt 2004; Karabaev & Skvortsova 2005; Sanner & Boutilier 2005; 2006), current formalisms are inadequate for specifying both factored actions and additive rewards in a fashion that allows reasonable scaling with the domain size. To remedy these deficiencies, we propose a novel extension of the FOMDP formalism known as a factored FOMDP. This representation introduces product and sum aggregator extensions to the FOMDP formalism that permit the specification of factored transition and additive reward models that scale with the domain size. We then generalize symbolic dynamic programming and linear-value approximation techniques to exploit product and sum aggregator structure, solving a number of novel problems in first-order probabilistic inference (FOPI) in the process. Having done this, we present empirical results to demonstrate that we can obtain effective solutions on the well-studied SYSADMIN problem whose complexity scales polynomially in the logarithm of the domain size—results that are impossible to obtain with any previously proposed solution method. Markov Decision Processes Factored Representation In a factored MDP, states will be represented by vectors ~x of length n, where for simplicity we assume the state variables x1, . . . , xn have domain {0, 1}; hence the total number of states is N = 2. We also assume a set of actions A = {a1, . . . , an}. An MDP is defined by: (1) a state transition model P (~x|~x, a) which specifies the probability of the next state ~x given the current state ~x and action a; (2) a reward function R(~x, a) which specifies the immediate reward